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ABSTRACT 


The  Pilot  Prediction  System  (PPS)  is  a  research  effort  designed  to  provide  Navy  managers  and  other  decision 
makers  with  improved  access  to  selection  and  training  data.  Marty  disparate  data  bases,  each  containing  partial  and 
sometimes  overlapping  information  on  selection  data  and  training  performance,  currently  exist.  There  has  been  no 
attempt  to  coordinate  the  bits  and  pieces  gathered  into  these  local  databases  into  a  coherent  whole.  Such  data  needs 
to  be  merged  and  the  anomalies  excised  so  that  a  more  global  picture  of  selection  data  and  training  performance 
can  emerge.  In  addition,  the  targeted  users  of  the  PPS  should  be  shielded  from  the  low-level  technicalities  of  the 
data  base  because  such  technical  details  are  of  no  concern  to  them.  For  the  same  reason,  the  statistical 
manipulations  that  provide  extrapolations  from  the  data  base  to  new  cases  can  be  hidden  from  view.  This  report 
documents  the  first  efforts  at  constructing  the  statistical  modeling  component  of  the  PPS  as  derived  from  Bayesian 
statistical  decision  theory.  It  enables  the  potential  user  of  the  PPS  to  predict  success  in  primary  flight  training  for 
flight  students  based  on  their  scores  on  selection  tests  and  ground  school  performance. 
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INTRODUCTION 


The  Pilot  Prediction  System  (PPS)  is  a  research  effort  designed  to  provide  Navy  managers  and  other  decision 
makers  with  improved  access  to  selection  and  training  data.  In  the  past,  our  collaborative  work  with  the  Chief, 
Naval  Fducatinn  and  Training  (CNET)  has  pinpointed  certain  deficiencies  in  providing  technical  support  for 
high-level  policy  questions  concerned  with  selection  and  training  issues. 

Many  disparate  data  bases,  each  containing  partial  and  sometimes  overlapping  information  on  selection  data 
and  training  performance,  currently  exist  The  data  in  these  databases  are  often  structured  according  to  different 
software  formatting  conventions.  The  proprietors  of  these  data  bases  are  geographically  dispersed  and  under  the 
control  of  different  administrators.  The  quality  of  the  data  varies  greatly.  There  has  been  no  attempt  to  coordinate 
the  bits  and  pieces  gathered  into  these  local  databases  into  a  coherent  whole.  These  disjunctive  data  need  to  be 
merged  and  the  anomalies  excised  so  that  a  more  global  picture  of  selection  data  and  training  performance  can 
emerge.  When  managers  need  to  make  a  decision  based  on  training  data  they  should  have  to  consult  only  one 
comprehensive,  error-free  data  base. 

In  addition,  the  targeted  users  of  the  PPS  should  be  shielded  from  the  low-level  technicalities  of  accessing  the 
data  base  as  well  as  the  statistical  manipulations  that  provide  extrapolations  from  the  data  base  to  new  cases.  One 
of  the  PPS’s  major  goals  is  to  provide  access  to  an  improved  data  base  in  a  “user-friendly”  manner.  One  analogy 
that  we  employ  quite  often  in  describing  the  PPS  is  that  it  should  function  much  like  a  typical  spreadsheet.  This 
would  allow  users  to  conduct  “what-if  ’  analyses  on  scenarios  of  their  choosing.  This  spreadsheet  would  operate  at 
the  user’s  desired  level  of  what  is  important  and  keep,  what  is  for  them,  the  unimportant  technical  operations 
hidden  from  view. 

A  primary  consideration  of  this  project  is  to  provide  statistical  models  that  allow  decision  makers  to  take  full 
advantage  of  the  available  data  in  order  to  make  predictions  about  the  future  success  of  an  individual.  A  map  or 
schematic  of  the  overall  selection  and  training  flow  would  be  shown  to  the  user  on  system  start-up.  The  potential 
user  of  the  system  need  only  indicate  via  a  mouse  click  which  available  data  connected  to  the  map  should  go  into  a 
model  as  predictor  variables.  Likewise,  a  mouse  click  would  indicate  which  criterion  variable  is  desired.  The 
appropriate  section  of  the  map  would  then  be  highlighted  to  show  the  user  what  portion  of  the  overall  schema  he 
has  chosen  to  investigate.  The  user  could  then  indicate  which  candidate  he  is  interested  in  looking  at.  The  system 
would  automatically  fetch  the  required  data  for  this  individual  from  the  data  base. 

The  PPS  would  link  these  choices  into  a  statistical  model.  The  numerical  routines  in  the  software  would  then 
perform  an  inference  indicating  the  predicted  outcome  for  this  individual  on  the  chosen  criterion  and  then,  most 
importantly,  also  provide  the  degree  of  confidence  that  the  user  might  have  for  this  prediction  by  the  system. 

For  example,  scores  on  the  various  subtests  of  the  Aviation  Selection  Test  Battery  (ASTB)  and  the  overall  Navy 
standard  score  from  Aviation  Preflight  Indoctrination  (API)  could  be  the  predictor  variables  of  interest  Graduation 
or  attrition  from  primary  flight  training  could  be  the  criterion  of  interest  Perhaps  a  student  comes  before  a 
Progress  Review  Board  because  of  substandard  performance  during  scheduled  early  familiarization  flights  in 
primary  flight  training.  Entering  the  student’s  Social  Security  Number  into  the  system  would  immediately  access 
his/her  ASTB  scores  and  overall  performance  during  API  from  the  data  base  and  enter  them  into  the  statistical 
model.  The  PPS  would  then  show  whether  this  student  was  a  predicted  pass  or  a  predicted  fail  from  primary  flight 
training.  In  addition,  the  confidence  level  of  this  prediction  would  be  indicated  by  giving  the  probability  of  this 
prediction  being  true.  This  student  might  be  a  predicted  pass  given  his/her  pattern  of  scores,  but  the  probability  of 
the  predicted  pass  would  be  only  60%.  A  threshold  confidence  level  might  have  been  established  at,  say,  an  80% 
confidence  level  with  the  result  that  this  student  might  be  moved  into  the  predicted  fail  category. 

In  this  fashion  we  envision  the  PPS  as  being  useful  to  the  Commodore  of  the  Training  Wing  where  this  student 
was  undergoing  training.  He  or  she  might  use  the  PPS  to  help  make  an  informed  decision  about  the  potential 
success  of  this  student  when  compared  to  the  success  profiles  of  the  students  already  in  the  data  base. 
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The  Observed  Data 

The  data  analyzed  in  this  report  were  provided  by  LT  Henry  Williams.  The  ASTB  test  scores  were  compiled  at 
the  Naval  Operational  Medicine  Institute  (NOMI),  and  the  API  scores  came  courtesy  of  the  Naval  Schools 
Command  (NASC),  both  at  NAS  Pensacola.  The  primary  flight  training  attrition  data  were  collected  from  Training 
Wing  5,  which  consists  of  three  training  squadrons,  VT-2,  VT-3,  and  VT-6,  located  at  Milton,  Florida.  The 
primary  flight  training  attrition  data  covered  the  time  span  from  June  1994  to  January  1998.  The  1,054  entries  in 
the  data  base  were  composed  exclusively  of  Navy  and  Marine  Corps  student  pilots.  Only  those  students  who  had 
taken  the  version  of  the  ASTB  as  revised  in  1992  were  included  in  the  data  base.  A  total  of  94%  of  the  students 
were  male  and  the  remaining  6%  female.  From  the  overall  total  of  1,054  students  in  the  data  base,  936  students 
passed  primary  flight  training  and  118  students  were  attrited  (i.e.,  failed  training). 

The  analysis  in  this  report  is  predicated  on  the  assumption  of  multivariate  normality  for  the  predictor  variables. 
This  section  lists  the  sufficient  statistics  for  this  assumption,  viz.,  the  sample  means  and  the  sample 
variance-covariance  matrix.  Table  1  presents  the  means  and  standard  deviations  for  the  five  predictor  variables 
studied  in  this  report.  MVT,  MCT,  SAT,  and  ANI  are  four  subtests  from  the  ASTB.  MVT  stands  for  Mathematics 


Table  1:  Means,  standard  deviations,  and  sample  sizes  over  the  five  predictor  variables  for  the  two  categories  of 
pass  and  fail  in  primary  flight  training. 


Test 

Mean 

Pass 

SD 

N 

Mean 

Fail 

SD 

N 

MVT 

27.18 

5.03 

936 

26.47 

4.79 

118 

MCT 

22.15 

3.70 

936 

20.36 

3.90 

118 

SAT 

27.79 

4.77 

936 

26.36 

5.28 

118 

ANI 

19.16 

3.31 

936 

17.85 

3.17 

118 

NSS 

52.01 

5.65 

935 

47.23 

6.53 

118 

and  Verbal  Test,  MCT  stands  for  Mechanical  Comprehension  Test,  SAT  stands  for  Spatial  Apperception  Test,  and 
ANI  for  Aviation  and  Nautical  Interest.  NSS  stands  for  Navy  Standard  Score,  a  standardized  score  for  overall 
performance  in  API.  These  five  scores  are  given  for  two  groups,  (1)  those  who  successfully  graduated  from 
Primary  flight  training,  and  (2)  those  who  attrited  from  Primary  flight  training  for  arty  reason.  These  groups  will 
be  called  the  PASS  group  and  the  FAIL  group  respectively.  The  sample  sizes  for  each  group  are  also  given  in 
Table  1.  The  attrition  rate  for  this  sample  is  seen  to  be  118/1054  (11.2%).  This  is  quite  typical  for  the  attrition 
rate  in  primary  training  which  historically  has  hovered  about  the  10%  mark.  The  mean  score  is  higher  for  the 
PASS  group  than  for  the  FAIL  group  on  all  five  predictor  variables.  The  standard  deviations  across  the  two  groups 
seem  to  be  roughly  comparable. 

Table  2  gives  the  sample  variance-covariance  matrix  for  the  PASS  group  while  Table  3  gives  the  same 
information  on  the  FAIL  group.  The  values  along  the  diagonal  of  each  matrix  give  the  sample  variances  of  the 
appropriate  variables,  i.e.,  the  square  of  the  standard  deviations  from  Table  1. 

Although  the  variance-covariance  matrix  is  appropriate  for  the  numerical  computations  described  later,  the 
sample  correlations  among  the  five  predictor  variables  are  more  informative  to  our  eyes.  Tables  4  and  5  exhibit  the 
pattern  of  correlations  for  the  PASS  group  and  the  FAIL  group,  respectively.  The  correlations  appear  to  be  quite 
similar  for  the  two  groups. 
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Table  2:  The  sample  variance-covariance  matrix  SpaBS  for  N  =  936  subjects  who  passed  primary  flight  training. 


Test 

MVT 

MCT 

SAT 

ANI 

NSS 

MVT 

25.335 

MCT 

8.375 

13.660 

SAT 

2.156 

4.204 

22.790 

ANI 

2.050 

4.252 

3.043 

10.930 

NSS 

9.760 

6.616 

3.822 

3.825 

31.947 

Table  3:  The  sample  variance-covariance  matrix  Spaii  for  N  -  118  subjects  who  failed  primary  flight  training. 


Test 

MVT 

MCT 

SAT 

ANI 

NSS 

MVT 

MCT 

15.188 

SAT 

-1.721 

5.399 

27.875 

ANI 

1.851 

3.756 

2.347 

10.079 

NSS 

11.967 

8.687 

6.095 

1.531 

42.691 

Table  4:  The  correlation  matrix  of  the  five  predictor  variables  for  the  N  =  936  subjects  who  passed  primary  flight 
training. 


Test 

MVT 

MCT 

SAT 

ANI 

NSS 

MVT 

1.000 

MCT 

.450 

1.000 

SAT 

.090 

.238 

ANI 

.123 

.348 

.193 

1.000 

NSS 

.343 

.317 

.142 

.205 

1.000 

Table  5:  The  correlation  matrix  of  the  five  predictor  variables  for  the  N  —  118  subjects  who  failed  primary  flight 
training. 
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Making  the  decision  on  a  new  candidate 

How  do  you  make  the  decision  to  predict  a  pass  or  a  fail  for  any  given  candidate  on  the  basis  of  the  observed 
scores  on  the  five  predictor  variables?  The  answer  to  this  question  is  to  use  Bayesian  Decision  Theory  (Berger  [1], 
Coombs,  Dawes  and  Tversky  [2]).  Within  this  approach,  we  try  to  find  that  decision  which  has  the  minimum 
expected  loss  with  respect  to  the  probability  of  passing  or  failing. 

Expectation  in  statistical  parlance  is  the  same  as  the  average,  with  the  average  defined  for  any  generic  discrete 
function  f(x)  as, 


£[/(*)]  =  £/fem*;)-  (1) 

3=1 

The  loss  corresponds  to  the  function  f(x).  It  depends  on  two  arguments:  (1)  what  actually  happened,  called  the 
state  of  nature,  0;  and  (2)  the  decision  taken.  The  decision  is  sometimes  called  the  action,  and  we  will  write  a* 
as  the  fcth  decision  taken  to  avoid  confusion  with  dk.  The  uncertainty  surrounds  which  state  of  nature,  6j,  will 
actually  occur  and  so  we  represent  this  uncertainty  with  a  discrete  probability  distribution,  P(0).  In  symbols,  the 
expected  loss  is  written  as, 


expected  loss  of  decision  ak  =  ^  4(0y,  ak)  P(0y).  (2) 

i=i 

The  current  situation  has  only  a  total  of  n  =  2  possible  true  states  of  nature, 

0i  =  actual  pass 
02  =  actual  fail 

and  two  possible  decisions  that  could  be  taken  under  each  state, 

a!  =  predict  pass 
tt2  =  predict  fail. 

There  are  four  possible  losses  corresponding  to  all  possible  combinations  of  the  two  decisions  and  the  two  states  of 

nature.  The  losses  for  each  decision  under  each  state  of  nature  are  represented  by  the  loss  matrix  shown  in  Table  6. 


Table  6:  The  loss  matrix  for  the  decision  problem  of  choosing  candidates  to  enter  primary  flight  training. 


Predict  Pass 

Predict  Fail 

ax 

a  2 

Actual  Pass 

*1 

i(0i,  ai) 

0 

4(0i,  a2) 

<?i 

Actual  Fail 

02 

4(02,  ax) 

c2 

4(02,  U2) 

0 

By  convention,  a  loss  of  0  is  assigned  to  the  two  correct  decisions,  i.e.,  predicting  that  a  candidate  will  pass 
when  he  or  she  actually  does  pass  training,  or  predicting  that  the  candidate  will  fail  when  she  or  he  actually  does 
fail  training.  A  placeholder  is  inserted  for  the  monetary  costs  incurred,  C\  and  C2,  for  the  two  ways  to  make  an 
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incorrect  decision,  that  is,  predicting  that  a  candidate  will  pass  when  he  or  she  actually  fails  training,  or  predicting 
fail  when  the  candidate  actually  passes  training. 

Inserting  these  costs  into  Equation  (2)  results  in 


expected  loss  ai  = 

L(0i,  ai)P(6i)  +  £(02,  ai)-P(02) 

(3) 

expected  loss  predicted  pass  = 

[0  x  P(Pass)]  +  [C2  x  P(Fail)] 

(4) 

= 

C2  x  P(Fail) 

(5) 

expected  loss  <12  = 

L(e1,a2)P(e1)  +  L(e2,a2)P(e2) 

(6) 

expected  loss  predicted  fail  = 

[Cl  X  P(Pass)]  4-  [0  x  P(Fail)] 

(7) 

= 

Ci  x  P(Pass) 

(8) 

The  decision  rale  itself  is  quite  simple.  If  the  expected  loss  of  the  predicted  pass  is  less  than  or  equal  to  the 
expected  loss  of  the  predicted  fail,  then  predict  pass,  otherwise  predict  fail. 

C2  x  P(Fail) 

<  Ci  x  P(Pass) 

(9) 

C2 

C ! 

^  P(Pass) 

“  P(Fail) 

(10) 

P(Pass) 

P(Fail) 

IV 

(11) 

Then 

Predict  Pass 

Else 

Predict  Fail 

Numerical  examples  of  decision  rule 

This  section  presents  some  numerical  examples  of  the  decision  rale  as  given  in  Equation  (11).  It  also  illustrates 
that  the  probability  of  passing  needed  to  predict  a  pass  will  change  with  the  costs  associated  with  the  incorrect 
decisions.  For  the  first  example,  let  the  cost  of  training  someone  who  later  fails  be  greater  than  the  cost  of 
rejecting  someone  who  would  have  passed  training.  Table  7  presents  an  example  of  such  a  situation 


Table  7:  The  loss  matrix  for  the  decision  problem  of  choosing  candidates  to  enter  primary  flight  training  when  it  is 
more  expensive  to  train  eventual  failures  than  to  reject  some  successful  c^ndidat^s 


Predict  Pass  Predict  Fail 
Ci  1  0>2 


£(0i,  ai) 

£(0i,  a2) 

Actual  Pass 

0 

$200,000 

£(02,  ai) 

£(02,  02) 

Actual  Fail  62 

$1,000,000 

0 

In  an  ideal  situation  these  costs  would  be  determined  through  a  detailed  economic  analysis  conducted  by 
experts  in  training  and  selection.  Unfortunately,  I  am  unfamiliar  with  any  such  analyses.  The  costs  used  in  the 
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following  examples  were  chosen  by  me  as  not  unreasonable  numbers  to  illustrate  the  mechanics  of  the  decision 
rule.  In  any  case,  the  purpose  of  this  report  is  to  provide  the  proper  quantitative  framework  for  making  decisions 
about  predicting  a  pass  or  a  fail  for  an  individual.  The  “correct”  cost  figures,  when  agreed  upon,  can  be  easily 
inserted  into  the  loss  matrix.  In  the  PPS,  the  main  utility  of  the  loss  matrix  may  be  in  allowing  users  to  assess  the 
effects  of  different  decision  thresholds  in  a  “what-if  ’  exercise. 

Suppose  that  the  probability  of  passing  primary  flight  training  based  on  the  scores  the  candidate  obtained  on 
the  five  predictor  variables  is  80%.  Therefore,  the  probability  of  failing  is  20%. 


P(Pass) 

> 

c2 

P(Fail) 

C ! 

Then 

Predict  Pass 

Else 

Predict  Fail 

P(Pass) 

.80 

P(Fail) 

.20 

= 

4 

C2 

C 

Ci 

O 

4 

5 

Therefore, 

Predict  Fail 

In  this  case,  a  probability  of  passing  equal  to  80%  is  simply  not  high  enough  to  commit  to  a  decision  to  predict  a 
pass  because  of  the  high  cost  of  the  wrong  decision  to  let  someone  into  training  when  they  fail.  If  a  candidate’s 
probability  of  passing  based  on  the  test  scores  were  raised  to  90%,  then 


P(Pass)  ^  C2 
P(Fail)  “  Ci 

and  the  candidate  would  be  admitted  into  training  as  a  predicted  pass. 

As  the  ratio  of  losses  for  these  incorrect  decisions  start  to  climb,  then  it  becomes  even  harder  for  a  candidate  to 
get  accepted  into  training.  As  the  following  numerical  example  illustrates,  an  increased  ratio  of  C2  to  Cx  raises  the 
threshold  for  acceptance  into  training  even  higher.  Table  8  presents  the  scenario  when  training  is  very  expensive, 
but  the  pool  of  qualified  applicants  wishing  to  be  trained  is  large.  In  this  altered  situation. 


P(Pass) 

P(Fail) 


>  15 


therefore,  the  probability  of  passing  must  be  around  94%  or  greater  for  the  decision  rule  to  recommend  acceptance 
into  training.  Going  back  to  first  principles,  this  means  that  the  average  loss  for  predicting  a  pass  for  arty 
probability  of  passing  less  than  94%  is  greater  than  the  average  loss  of  predicting  a  fail. 


Of  course,  the  change  in  the  decision  criterion  can  work  just  as  well  in  the  opposite  direction.  If  the  losses  for 
the  two  incorrect  decisions  were  judged  to  be  of  equal  value  as  in  Table  9  below,  then  the  ratio  of  costs  would 
change  to 
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Table  8:  The  loss  matrix  for  the  decision  problem  of  choosing  candidates  to  enter  primary  flight  training  when 
training  is  very  expensive  but  the  pool  of  qualified  applicants  is  large. 


Predict  Pass 

Predict  Fail 

ax 

a2 

Actual  Pass 

0i 

L(6i,ai) 

0 

L(0 1,0-2) 
$100,000 

Actual  Fail 

02 

1.(02,  Ol) 

$1,500,000 

1.(02,  O-2) 

0 

As  soon  as  the  ratio  of  the  probability  of  passing  to  the  probability  of  failing 

P(Pass) 

P(Fail) 

becomes  greater  than  1,  a  predicted  pass  results.  A  probability  of  passing  equal  to  50%  or  greater  would  be 
sufficient  to  predict  a  pass  in  this  case  . 


Table  9:  The  loss  matrix  for  the  decision  problem  of  choosing  candidates  to  enter  primary  flight  training  when  it  is 
equally  as  bad  to  reject  someone  who  could  have  passed  as  it  is  to  admit  someone  who  fails 


Predict  Pass  Predict  Fail 

O-l  02 


L(6i,ax) 

1/(01,  O2) 

Actual  Pass  0\ 

0 

$500,000 

1.(02,  ai) 

1.(02,  a2) 

Actual  Fail  02 

$500,000 

0 

The  derivation  of  the  posterior  odds 

Except  for  the  potential  subjective  or  economic  difficulty  in  assigning  the  costs,  Ci  and  C2,  the  calculation  of 
this  part  of  the  decision  rule  is  trivial,  unlike  that  of  the  calculation  of  the  ratio  of  probabilities.  A  great  degree  of 
non-trivial  mathematics  must  be  brought  to  bear  in  that  case.  Fortunately,  the  difficult  part  of  the  mathematics  has 
already  been  worked  out  by  Press  [3]  and  Geisser  [4],  In  this  section,  we  present  the  general  outline  for  the 
formula  of  the  ratio, 

P(Pass) 

P(Fail) 

as  needed  for  the  decision  rule.  The  ratio  is  commonly  known  as  the  “posterior  odds.”  The  formula  used  in  the 
computer  program  to  calculate  the  numerical  results  is  also  given  at  the  end  of  this  section. 

Actually,  P(Pass)  or  P(Fail)  is  too  simplistic  a  notation  for  what  is  really  a  probability  based  on  considerable 
information.  That  information  consists  of  the  scores  obtained  on  the  five  predictor  variables  by  the  candidate  we 
wish  to  classify.  Thus,  let  Dn+l  stand  for  these  scores  for  the  “n  +  1st”  subject,  i.e.,  the  new  oan&iate  we  wish  to 
classify.  Then  Dn  is  the  notation  for  the  data  in  the  database  from  the  n  students  who  also  have  scores  for  the 
predrctor  variables  and  for  whom  we  know,  as  well,  their  success  or  failure  in  primary  flight  training. 

Therefore,  we  now  employ  the  more  accurate  notation  of 

P(Pass|Z>n+1)  and  P(Fail|Z>n+1) 


7 


as  the  posterior  classification  probability  for  passing  and  failing.  By  Bayes’s  Formula  these  two  posterior 
probabilities  are  written  as  a  function  of  the  likelihood  and  the  prior  probabilities, 


P(Pass|£>n+i) 


_ P(Dn+i|Pass)  P(Pass) _ 

P(Dn+i|Pass)  P(Pass)  +  P(Dn+1|Fail)  P(Fail) 


(12) 


P(Fail|£>„+1) 


_ P(Dn+i|Fail)  P(Fail) _ 

P(D„+i|Pass)  P(Pass)  +  P(£>„+i|Fail)  P(Fail) 


(13) 


In  forming  the  posterior  odds,  we  can  cancel  out  the  denominator  in  the  posterior  probability  of  both  terms.  The 
posterior  odds  now  can  be  written  as  a  ratio  of  the  likelihood  times  the  prior  probability  for  passing  and  failing, 


P(Pass|Dn+i)  _  P(Dn+i  |Pass)  P(Pass) 
P(Fail|£>„+1)  “  P(A»+i  |Fail)  P(Fail) 


For  an  introduction  to  the  Bayesian  approach  as  used  in  scientific  inference,  see  Blower  [5]. 


So  far  the  development  has  been  relatively  straightforward,  but  now  the  difficult  part  of  the  mathematics 
intrudes.  The  likelihood  terms  in  Equation  (14)  are  called  predictive  densities  in  the  Bayesian  approach.  The 
predictive  density  for  either  one  of  our  two  classification  categories  should  be  written  as 

P(A>+ 1  |Pass,  Dn)  or  P(Dn+ 1  [Fail,  £>„) 

showing  the  dependence  on  the  past  data,  Dn.  The  predictive  density  marginalizes  over  all  the  parameters  used  in 
the  assumptions  for  how  the  scores  are  generated.  In  this  case,  the  assumption  is  multivariate  normality  for  the 
scores  with  two  parameters,  fx  and  E,  where  /x  is  the  vector  of  population  means  for  the  scores  and  E  is  the 
population  variance-covariance  matrix  of  the  scores.  Marginalizing  over  these  two  parameters  to  find  the  predictive 
density  yields, 

P ( Dn+ 1 [Pass,  Dn)  =  J' J  P(Dn+  i|/x,  E,Pass,  Dn)  P(ti)  EjPass,  Dn)  dfXpa3S  dSpatg  (15) 

This  is  the  product  of  the  likelihood  of  the  scores  for  the  new  candidate  and  the  posterior  probability  of  the 
parameters  with  this  product  taken  over  all  values  of  the  parameters. 


Skipping  over  all  the  goiy  mathematical  manipulations  of  Equation  (15)  (see  Press  [3],  pp.  153-155  for  the 
details),  we  show  only  the  final  result.  The  predictive  probabilities  for  the  scores  of  a  new  candidate  (£>„+ 1)  given 
the  old  data  (Dn)  and  a  particular  category  (Pass,  Fail)  are  shown  first  in  a  general  outline  form  and  then  in 
explicit  detail. 


P{Dn+ i  |Pass,  Dn)  =  term  1  x  term  2 

(16) 

P(Dn+i  |Fail,  Dn)  =  term  3  x  term  4 

(17) 

Posterior  Odds  =  j?*+1J 

P(Fail|Dn+1) 

(18) 

P(Dn+i|Pass,  Dn)  x  P(Pass) 

P(Dn+i  |Fail,  Dn)  x  P(Fail) 

(19) 

term  1  x  term  2  P(Pass) 
term  3  x  term  4  P(Fail) 

(20) 

For  constructing  the  computer  program  to  calculate  the  posterior  odds,  the  detailed  formulas  follow.  Terms  1  and  3 
are  patterned  alike  as  are  terms  2  and  4. 

term  1  =  |  (NFail  -  1)  SFail\1/2  T  T  NPaiS(NFail  +  If'2  (21) 
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[/  Nip  -I  \  -l^Fail/Z 

1  +  °L  2  J  (A»+l  -  ®Fo«)T  •S'f'iji  (Ai+l  -  XFail) 
term  3  =  |(iW  -  1)  5fas,|1/2  T  T  - P)  NFail(NPass  +  l)p/2 

[/  7VT  \  I  Npassl 2 

1  +  (jVg"?-lJ  (£>n+l-^PaSS)T  Sfi..  (A.+  1  ~  *Pa„) 

Table  10  briefly  describes  the  symbols  used  in  Equations  (21)  through  (24). 

Table  10:  Explanation  for  symbols  in  Equations  (21)  through  (24). 


(22) 

(23) 

(24) 


Symbol 

Explanation 

P(Pass) 

P(Fail) 

Prior  Probability  of  Passing  or  Failing 

N Pass 

N Fail 

Sample  sizes  of  Pass  and  Fail  groups 

p 

Number  of  predictor  variables 

xPass 

&Foil 

Sample  means  on  p  tests 

Spass 

Spail 

Sample  covariance  matrices  for  p  tests 

Dn+\ 

Scores  on  predictor  variables  for  new  candidate 

{Dn+1  •EPass')'^ 

(-Dn+1  Fail  )^~ 

Transpose  of  difference  between  vector  of  new  score  and 
sample  means 

\Spass\ 

\SFail\ 

Determinant  of  sample  covariance  matrices 

Spi,s 

SfIu 

Inverse  of  sample  covariance  matrices 

W) 

Gamma  function  with  N  as  argument 

Statistical  Decision  Theory 

Before  beginning  the  numerical  exercises,  we  would  like  to  express  the  equations  just  derived  in  a  form 
compatible  with  Bayesian  statistical  decision  theory.  The  decision  algorithm  from  statistical  decision  theory  can  be 
stated  in  the  following  simple  terminology.  If  the  likelihood  ratio  is  greater  than  some  response  threshold,  then 
commit  to  a  particular  decision.  In  symbols. 

If  C(x)  >  0  then  predict  appropriate  outcome 
Equation  (11),  as  the  ratio  of  the  posterior  odds,  should  now  be  correctly  written  as 

If  -  7T  then  Predict  Pass  otherwise  Predict  Fail  (25) 

P(Fatl|Z3n+i)  Ci  v  ’ 


Substituting  Equation  (19)  for  the  posterior  odds  yields, 


P(Dn+i|Pass,  Dn)  x  P(Pass) 

>  ^ 

(26) 

■P(Ai+i|Fail,  Dn)  *  P(Fail) 

-  <?r 

P(Dn+i\?ass,Dn) 

C2 

>  _ _  y 

P(Fail) 

(27) 

P(£>n+i|Fail,D„) 

-  Cl 

P(Pass) 

The  left  hand  side  of  Equation  (27)  is  in  the  form  of  a  likelihood  ratio  since  it  is  the  ratio  of  the  probability  of 
the  data  from  a  candidate  given  the  old  data  and  the  Pass  group  over  the  probability  of  the  data  from  a  candidate 
given  the  old  data  and  the  Fail  group.  That  is, 

C(x)  =  £(data  from  candidate) 

The  right  hand  side  of  Equation  (27)  is  a  function  of  the  costs  of  making  correct  and  incorrect  decisions  and  the 
prior  odds  of  failing  over  passing.  Together  they  make  up  0,  the  response  threshold.  Equation  (27)  therefore 
represents  a  decision  algorithm  in  the  form  of 

If  C(x)  >  0  then  predict  Pass 


Numerical  computations 

Numerical  examples  of  the  Baysian  decision  rule  as  derived  in  the  previous  sections  are  presented  here.  Table 
11  shows  the  scores  obtained  on  the  five  predictor  variables  for  three  new  candidates  for  whom  we  would  like  a 
classification  as  a  predicted  pass  or  a  predicted  fail.  The  criterion  for  such  a  prediction  will  be  to  minimize  the 
monetary  loss  experienced  with  an  incorrect  decisioa  For  this  first  set  of  examples,  we  arbitrarily  set  the  ratio 
C2/C1  equal  to  5,  merely  as  a  reasonable  supposition  about  costs.  For  example,  an  incorrect  rejection  of  a 
candidate  might  cost  $100,000,  and  permitting  an  eventual  attrition  into  primary  flight  training  might  cost 
$500,000.  Again,  the  five  predictor  variables  are  the  scores  on  four  subtests  of  the  ASTB,  the  MVT,  MCT,  SAT, 
and  ANI,  and  the  final  composite  grade  from  API,  NSS.  According  to  historical  records,  it  is  known  that  the 
probability  of  passing  primary  flight  training  given  the  current  selection  standards,  but  using  no  additional 
information  as  we  are  doing  here,  is  about  90%.  Therefore,  we  will  set  the  prior  probability  of  passing  at  90%  and 
the  prior  probability  of  failing  at  10%.  The  posterior  classification  probabilities  are  calculated  by  using  Equations 
(21)  through  (24).  The  sample  means  are  taken  from  Table  1  and  the  sample  covariances  from  Tables  2  and  3.  The 
sample  size  for  the  Pass  group  is  Npass  =  936  and  the  sample  size  for  the  Fail  group  is  Npaii  =  118.  The 
number  of  predictor  variables,  p,  is  equal  to  5.  The  D„+i  are  the  scores  listed  in  the  five  columns  of  Table  11 
after  the  Candidate  column. 

Table  11:  Examples  of  Bayesian  decision  rule  for  various  scores  obtained  by  three  candidates.  The  loss  ratio  for 
this  example  is  set  at  C2/C1  =  5. 


Candidate 

MVT 

MCT 

SAT 

ANI 

NSS 

P(Pass|Dn+1) 

Post.  Odds 

Decision 

1 

27 

22 

27 

mm 

52 

.9413 

WMM 

Predicted  Pass 

2 

26 

20 

26 

1 m 

47 

.8744 

mu 

Predicted  Pass 

3 

24 

18 

23 

15 

40 

.6608 

MEM 

Predicted  Fail 

The  first  candidate  scored  close  to  the  sample  mean  of  the  936  students  who  passed  primary  flight  training.  It 
is  not  surprising,  therefore,  that  the  posterior  classification  probability  is  quite  high  (at  about  94%)  that  this 
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candidate  will  pass  training.  Even  so,  a  high  probability  of  passing  training  is  not,  in  and  of  itself,  enough  to  malcp 
a  decision  to  predict  a  pass.  For  this  candidate  though,  the  posterior  odds  of  16.037  are  greater  than  5  so  we  can 
safely  predict  pass  given  our  cost  structure.  The  second  candidate  had  lower  scores  on  all  five  predictor  variables 
and  the  associated  posterior  classification  probability  is  correspondingly  lower  at  around  87%.  However,  the 
posterior  odds  for  this  candidate  are  still  greater  than  5,  so  we  again  predict  a  pass.  Had  the  cost  structure  been 
different,  say  Ca/  C\  =  10,  in  order  to  emphasize  the  importance  of  admitting  into  training  only  those  with  a  very 
high  probability  of  success,  then  a  different  decision  would  be  warranted  for  this  candidate  The  third  candidate 
achieved  even  lower  scores  to  the  point  where  the  posterior  classification  probability  of  passing  is  lowered  to  66%. 
Now  the  posterior  odds  drop  below  the  threshold  of  5,  and  the  decision  is  made  to  predict  that  this  will 

fail. 

Effect  of  varying  parameters 

It  is  easy  to  examine  the  effect  on  the  probability  of  passing  primary  flight  training  by  changing  arty  of  the 
parameters  of  the  quantitative  model.  An  example  is  provided  of  how  this  can  be  done  by  making  the  following 
changes: 

[1]  decreasing  the  size  of  the  number  of  students  who  passed  and  failed  training 

[2]  changing  the  prior  probability  of  passing  and  failing 

[3]  making  the  tests  more  discriminatory  by  widening  the  gap  in  the  sample  means 

[4]  increasing  the  correlation  among  the  predictor  variables. 

These  four  changes  are  operationalized  within  the  quantitative  model  by  setting  the  values  of  the  following 
parameters: 


[1]  NPass  =  400  and  NFail  =  100 

[2]  P(Pass)  =  .80  and  P(Fail)  =  .20 

[3]  xPass  =  (28, 23, 28, 20, 55)  and  xFait  =  (25, 19, 24, 16, 45) 

[4]  increasing  the  off-diagonal  elements  of  SPass  and  SFau. 

The  effect  of  changing  the  cost  structure  on  incorrect  decisions  has  already  been  discussed,  therefore  this  facet 
of  the  decision  rule  will  be  kept  the  same  as  in  the  last  numerical  example  where  Ca/Ci  =  5.  The  numerical 
results  from  the  changes  just  discussed  are  given  in  Table  12.  Like  Table  11,  Table  12  gives  the  scores  of  three 

Table  12:  The  effect  on  the  probability  of  passing  and  the  decision  to  predict  pass  or  fail  from  changing  parameters 
in  the  quantitative  model. 


Candidate 

MVT 

MCT 

SAT 

ANI 

NSS 

.P(Pass  |£)n+i) 

Post.  Odds 

Decision 

1 

28 

23 

28 

20 

55 

.9747 

38.487 

Predicted  Pass 

2 

26 

20 

25 

17 

47 

.5109 

1.045 

Predicted  Fail 

3 

28 

23 

28 

20 

45 

.6088 

1.556 

Predicted  Fail 

candidates  on  the  five  predictor  variables.  As  in  the  last  example,  the  fust  candidate  scores  at  the  mean  of  the  pass 
group,  but  since  the  sample  means  are  further  apart  than  in  the  first  example,  these  scores  are  now  more 
compelling  evidence  that  this  candidate  will  pass.  The  posterior  odds  are  38.487:1  as  opposed  to  16.037:1  for  the 
first  example.  The  second  candidate  scores  better  on  all  tests  in  that  he  is  above  the  mean  of  the  Fail  group 
whereas  his  counterpart  of  the  first  example  was  only  at  the  mean  of  the  Fail  group.  This  candidate  is  a  predicted 
fail  whereas  the  candidate  in  the  first  example  was  a  predicted  pass.  Again,  this  is  because  the  means  are  more 
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widely  separated,  with  the  result  that  this  second  test  batteiy  is  more  discriminating.  Another  contributing  factor  to 
the  predicted  fail  is  the  lower  prior  probability  of  a  pass  in  this  example. 

A  curious  phenomenon  is  revealed  by  the  third  candidate.  For  the  sake  of  discussion,  this  candidate  happens  to 
be  female.  She  scores  exactly  the  same  as  the  first  candidate  on  the  four  ASTB  tests,  but  is  at  the  mean  of  the  fail 
group  for  the  API  score.  The  first  candidate  was  a  confident  predicted  pass,  but  this  candidate  is  an  equally 
resounding  predicted  fail.  What  happened  here?  Is  the  result  of  just  one  test  score  so  detrimental  to  this 
candidate’s  prospects? 

The  answer  lies  in  the  nature  of  the  changed  covariance  matrices.  The  off-diagonal  elements  of  the  covariance 
matrices  for  both  the  pass  and  fail  groups  were  made  larger  to  reflect  a  situation  where  the  correlation  among  all 
five  tests  was  extremely  high.  For  example,  the  true  correlation  between  MVT  and  MCT  was  set  at  .90,  and  the 
true  correlation  between  MVT  and  NSS  was  set  at  .70.  The  pattern  of  intercorrelations  for  the  first  four  scores 
with  the  fifth  score  demands  a  score  close  to  the  mean  of  the  pass  group  to  fit  the  profile  of  a  pass  candidate.  The 
discordant  score  on  the  last  predictor  variable  is  enough,  given  the  extremely  high  intercorrelations,  to  make  her 
much  less  like  the  typical  pass  candidate.  The  posterior  odds  are  correspondingly  lowered  and  given  the  costs  of 
the  various  decisions  make  her  a  predicted  fail.  The  fact  that  the  prior  probability  of  a  pass  was  lowered  did  not 
help  this  candidate  either. 

Another  way  of  looking  at  this  situation  of  the  third  candidate  is  in  terms  of  the  number  of  pieces  of 
independent  information.  In  the  ideal  case,  there  would  be  no  correlation  among  the  five  predictor  variables  and 
the  off  diagonal  elements  of  the  sample  variance-covariance  matrices  would  be  small.  Each  of  the  five  test  scores 
would  then  be  providing  five  independent  sources  of  information  about  the  correct  category  the  candidate  belongs 
to.  In  the  situation,  however,  of  large  off-diagonal  elements  in  the  sample  variance-covariance  matrices,  the  test 
scores  are  highly  correlated.  There  are  not  really  five  pieces  of  independent  information.  For  the  third  candidate 
we  could  say  roughly  that  there  are  only  two  pieces  of  independent  information,  one  indicating  she  belongs  to  the 
pass  group  and  the  other  indicating  that  she  belongs  to  the  fail  group.  So,  on  this  viewpoint,  it  is  not  surprising 
then  that  her  probability  of  passing  is  driven  down  towards  50%. 

Summary 

The  high-level  policy  decisions  that  depend  on  access  to  selection  and  training  data  could  be  improved  by  a 
more  coordinated  effort  at  upgrading  data  bases  and  integrating  statistical  prediction  models.  NAMRL  believes  that 
the  Pilot  Prediction  System  is  a  research  effort  that  will  move  us  closer  to  that  goal. 

This  report  outlines  some  initial  thoughts  on  the  architecture  of  the  PPS  and  how  it  might  help  its  targeted 
audience.  More  specifically,  we  have  presented  in  some  detail  a  statistical  model  that  predicts  whether  a  student 
will  pass  or  fail  in  primary  flight  training  as  a  function  of  four  selection  test  scores  and  overall  achievement  in 
API.  It  not  only  deals  with  this  kind  of  quantitative  information  in  an  optimal  fashion  through  the  Bayesian 
assessment  of  the  probability  of  passing,  but  also  takes  into  account  the  judgmental  or  economic  cost  factors 
involved  in  making  correct  and  incorrect  decisions. 

A  computer  program  has  been  written  to  implement  the  Bayesian  posterior  classification  probability  as  well  as 
the  decision  theory  aspects.  Some  numerical  examples  were  presented  to  show  how  the  program  works  to  predict 
success  for  candidates  who  have  not  yet  entered  primary  flight  training,  but  for  whom  we  have  scores  on  five 
predictor  variables.  Such  a  program  can  serve  as  one  core  module  in  a  user  friendly,  spreadsheet-like 
implementation  of  the  PPS.  The  PPS  implementation  philosophy  is  to  shield  its  users  from  technical  issues  such  as 
using  statistical  prediction  models  to  extrapolate  from  success  profiles  in  a  data  base.  This  feature  permits  them  to 
concentrate  more  fully  on  the  larger  manpower  and  training  issues  that  beset  naval  aviation. 
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